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ABSTRACT 

This paper outlines one approach for adapting the 
widely used word association technique for use as an evaluation 
measure. In this approach, project participants are presented with a 
number of stimulus words reflecting project objectives and are asked 
to give free associations to these terms. Responses are scored either 
by the nature of semantic content or by judgment of subjective 
quality and the resulting scores for pre and post administrations are 
statistically compared. The technique has been successfully employed 
in several evaluation studies, and has premise of providing the 
evaluator with an instrument that has rapidly alterable stimuli. 
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This paper outlines one approach for adapting the widely used word 
association technique for use as an evaluation measure. In this approach, 
project participants are simply presented with a number of stimulus words 
reflecting project objectives and are asked to give free associations to these 
terms* Responses are scored either by the nature of semantic content or by 
judgment of subjective quality and the resulting scores for pre and post 
administrations are statistically compared. The technique has already been 
successfully employed in several evaluation studies with widely differing 
objectives and subjects, and has promise of providing the evaluator with an 
instrument that has rapidly alterably stimuli, allowing immediate reaction to 
changing project objectives. 
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ADAPTING WORD ASSOCIATION 1 FOR USE AS AN EVALUATION TECHNIQUE 

Word association has long been a popular technique in rlinicil 
and experimental investigations. It was originally used as a cllnicaJ 
technique for eliciting responses relating to underlying but unrecognized dynamic- 
structures. Later, the publication of a normative list of 100 words and their 
associates by Kent and Rosenoff (1910) provided a comparison point for many 
descrif -e and methodological studies of verbal behavior. More recent studies 
with word association have focused on the experimental investigation of verbal 
structures and verbal development. The technique has proven useful in group 
comparison studies and in experimental manipulations of factors influencing 
verbal associations. 

The studies presented in this paper extend the word association technique 
into the area of educational evaluation. The rationale for this extension 
is that word association represents a low-level but direct means of assessing 
the degree to which an individual benefits cognitively from an educational 
experience. In simple terms, it is assumed that if an individual has gained 
anything cognitively from exposure, it will be reflected in the quality and 
frequency of associations evoked by elements from the material or situation 
experienced. Past research has suggested that word associations are stable 
(Jung, 1962) and yet are quite sensitive to experimental manipulation, which 
provides the potential for use as an evaluation technique. 

From the outset, it must be pointed out that the technique presented here 
is not presumed to assess all types of outcomes. These applications have 
concentrated on the cognitive aspects of behavior. While word association 
may well also be an indicant of non-cognitive outcomes, we have not investigated 
this possibility. Further, we are not recommending that this approach should 
replace other, stronger evaluation techniques-such as criterion-referenced tests- 
Egfc Wh6n SUCh techni( J ues are appropriate. Rather, we believe that word association 

can rlay a valuable role in evaluations where stroncer tachninu** *r a „„► , 



or, due to practical limitations, wilJ not be applied. 

The word association approach bfung presented here has been used in three 
types of studies to date. It was fi"st used in a Title 1 summer enrichment 
program, where culturally deprived students were given the opportunity to 
visit a number of historical, cultural, business, civic and recreational sites 
(DiLuzio and Katzenmeyer, 1970; DiLizio, Lesyk and Blixt, 1971; Hunter and Blixt, 
1972) ♦ It has also been used in a Title III K-6 curriculum development study of 
technology and its impact on society (Blixt and DiLuzio, 1972). Finally, it was 
employed in an experimental A;ollege unit on war and peace (Deming, 1971). 

Methodology 

We bave employed fie word association technique in diverse evaluations , 
with details of the procedures differing somewhat in each according to need. The 
approach presented here is currently being employed in the Title III evaluation 
mentioned above. 

Selection of Stimuli and Administration 

In th^ technology study, the teacher for each unit is asked to supply 
a number of words drawn from the content of the unit to be presented. These 
words are examined by the evaluator for linguistic structure, meaningf ulness , 
(Paivio, Yuille, and Madigan, 1968) and associative value (Palermo and Jenkins, 
1964). Then in cooperation with the teachers, five words are chosen for the evaluation. 
These five words are presented to the children prior to the introduction of the 
curriculum unit and the children are instructed to write down as many associations 
to each as possible in one minute. For younger children the associations are 
given orally and recorded by the teacher. After completion of the curriculum 
unit, the came five words are again presented to the students under the same 
instructions. Thus, this assessment procedure takes approximately ten minutes 
of the students 1 time for each unit. 
Scoring 



£J^£ Again, several approaches have been tried in scoring the word associat 
For younger students, responses are currently being scored for n„^, 



ions . 
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3 point scale: 3 for a synonym or superordinate; 2 for a major use or major 
characteristic; and 1 for a minor use or minor characteristic. Examples of 
this scoring procedure are given in Appendi:: A. The frequency of relevant 
responses is also recorded . 

When employing the technique in the evaluation of the experimental college 
class, the quality of associations was determined by content experts. Four 
Political Scientists reached grouo consensus for each response given. As yet 
we have not developed an objective scoring scheme for college students. 
Scorer Reliability 

In most of our investigations of the word association technique, scorer 
reliability has been no problem as the scorers were members of the evalua:ion 
team. Under circumstances of extensive interaction among team members we 
have found that nearly perfect agreement among judges can be reached qu^te 
rapidly. However, it is our intention that the technique will also be useful 
for project personnel and teachers and will not require the presence of a tiained 
evaluator. For that reason, we have had teachers in the Title III Project do 
scoring for their own units. Interscorer reliabilities for teachers and the 
evalualor are given in Table 1. These results are reasonably good and notable 
because they were achieved with minimal training of the teachers. It is expected 
that reliabilities would be increased with additional training. 

Validity 

Evidence for validity of this technique is still preliminary and incomplete; 
however, we have collected data in several of the studies that have demonstrated 
its sensitivity to change after instruction. Results from the Title I summer 
program are given in Tables 2 and 3. The number of significant increases in 
these tables is striking in light of the fact that the "instruction" presented 
was simply a trip to an historical, civic, business, or recreational setting, 
and that the pre and post evaluations occurred only three days apart. Teachers 
on this project were also asked to rank the trips for degree of educational worth. 
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Table 1 

Inter-scorer Reliabilities for Teachers and Evaluator 
in Title III Technology Curriculum Project, Spring, 1972* 



Teacher A B 


C 


D 


Evaluator 


A .782 


.691 


.752 


.861 


B 


.802 


.671 


.853 


C 




.782 


.761 


D 






.856 



Evaluator 



*Each scorer scored responses of 25 children to 5 concepts in each 
of grades K, 2 and 4. 



Table 2 



Pre and Post Test Comparison of Word Association 
Quality Scores for Students Participating in 
Title I Field Trips 

Summer, 1971 



Field Trip 


df 


Pretest 
Mean 


Posttest 
Mean 


Mean 

Difference 


t 


Proba- 
bility 


Dover 


39 


2.626 


2.689 


.063 


.4998 




Dairy 


25 


3.659 


3.804 


0.145 


1.4403 




Fruit Farm 


31 


2.798 


2.976 


.177 


2.3435 


.05 


Wooster 


32 


1.399 


2.491 


1.091 


7.3171 


.01 


Sea World 


23 


2.235 


2.220 


-.015 


-.1086 




Museum 


20 


3.391 • 


3.356 


-.035 


-.3358 




Court 


15 


3.195 


3.372 


.177 


1.0669 




Pittsburgh 


23 


2.625 


2.632 


.006 


.0546 




Playhouse 


29 


2.513 


2.862 


.709 


6.6314 


.01 


Columbus 


24 


1.500 


2.954 


1.454 


8.4634 


.01 


Niagra Falls 


26 


2.121 


2.229 


.109 


.8370 




Airport 


22 


3.047 


3.317 


.270 


1.4632 




Cooks Forest 


20 


3.227 


3.149 


-.078 


-.6206 




Akron 


25 


1.966 


2.778 


.813 


5.3593 


.01 
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Table 3 • 

Pre and Post Test Comparisons of Word Association 
Frequency Scores for Students Participating in 
Title I Field Trips 
Summer, 1971 





Field Trip 


df 


Pretest 
Mean 


Posttest 
Mean 


Mean 

Difference 


t 


Proba- 
bility 


Dover 


39 


12.850 


17.150 


4.300 


7.9849 




Dairy 


25 


18.769 


18.885 


.115 


.1031 




Fruit Farm 


31 


20.656 


24.969 


4.313 


4.6239 


01 


Wooster 


32 


5.727 


15.879 


10.152 


9.0584 


.01 


Sea World 


23 


18.042 


22.917 


4.875 


4.003] 


.01 


Museum 


20 


20.095 


23.095 


3.000 


2.1163 


.05 


Court 


15 


17.562 


20.437 


2.975 


3.3986 


.01 


Pittsburgh 


23 


16.417 


22.958 


6.542 


4.8750 


.01 


Playhouse 


29 


14.800 


21.933 


7.133 


6.0450 


.01 


Capitol 


24 


12.160 


23.280 


11.120 


6.9440 


.01 


Niagara Falls 


26 


15.111 


17.444 


2.333 


2.0888 


.05 


Airport 


22 


21.522 


23.957 


2.435 


1.2639 




Cooks Forest 


20 


25.714 


32.238 


6.524 


3.6398 


.01 


Akron 


25 


12.385 


19.115 


6.731 


8.3622 





-7- 

This ranking correlated .82 with mean word association quality scores for 
those trips. 

Data from the Title III Project on technology are given in Table 4. Results 
here are less striking than with the Title I Project, although there were a number 
of significant difference*? between the experimental and control groups in adjusted 
post-test associations. However, it should be pointed out that these curriculum 
units were being presented for the first time and that the word association technique 
was the only one of a number of evaluation instruments that showed consistent 
differences between experimental and control groups favoring the experimental groups. 

Significant differences were also found in the experimental college study. 
There the experimental group had higher .quality of association scores than the 
control group across five concepts. No significant differences were found for 
frequency of relevant associations. 

Several correlation studies hav<" also been carried out in the Title III 
Project. Intercorrelations of post-test word association responses and 
subscales of math, f 'dv skills, 'ind reading from the California Achievement 
Test are given in T >. It is apparent that the individual differences 

found in this word association task are strongly related to reading 
achievement and to some extent with study skills. Intercorrelations 
of word association quality scores and teachers ranking of each child f s knowledge 
of the unit are listed in Table 6. There was a sizeable relationship in all 
grades, but particular in the younger grades, between the individual differences 
variation for word association and the teachers 1 judgment of degree of knowledge* 
More important, however, is that the technique be sensitive to cognitive change, 
as the studies above suggest it is. 

Several additional validity studies are now underway. Word association 
scores will be correlated with teachers ranking of amount of knowledge gain, 
to give a sharper estimation of the extent to which t.he word association technique 
mirrors changes seen by the teachers. Second, the word association scores will be 

ERIC 

correlated with the child f s estimate of the degree to which he or she has benefited 



Table A 



Analysis of Covariance of Word Association Scores 
For Experimental and Control Students in a Title 
III Technology Curriculum Project, Grades K-6 

Spring, 1972 



Grade Wd. Assoc . df F Probability 



K 


Quality 


46 


18.11 


.01 




Frequency 


46 


12.54 


.01 


1 


Quality 


40 


3.56 






Frequency 


40 


0.00 




2 


Qua1 ity 


28 


8.61' 


.05 




Frequency 


28 


3.89 




3 


Quality 


28 


.17 






Frequency 


28 


1.85 




4 


Quality 


28 


2.03 






Frequency 


28 


13.31 


.01 


5 


Quality 


34 


.01 






Frequency 


34 


4.45 


.05 


6 


Quality 


22 


6.03 


.05 




Frequency 


22 


7.46 


.05 
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Table 5 

Intercurrclations o f Word Association 
and California Achievement Test Scores for 
4th, 5th and 6th Grades 



California Word 



Achievement 


Association 


4th 


5 th 


6 th 


Total 


Math 


Quality 


.04 


.22 


.25 


.08 




Frequency 


.01 


.07 


.11 


.06 


Study Skills 


Quality 


.22 


.20 


.37 


.34 




Frequency 


.19 


.18 


.21 


.05 


Reading 


Quality 


.37 


.41 


.66 


.56 




Frequency 


.23 


.38 


.36 


.61 



Table 6 

Correlations of Word Association Quality Scores 
and Teachers Rankings of Amount of Knowledge 



Level Mean Correlation Range 



K .77 .69-. 81 

1 .65 .52-. 72 

2 .60 .43-. 75 

3 .44 .33-. 59 

4 .57 .41-. 69 

5 .41 .55-. 61 

6 .35 .21-. 44 



*The correlations of association quality score and the teacher's 
ranking was done for each of 5 concepts at each grade leve 1 . . 



by having the children designate the associations lhat they conside 
the best, to find if our scoring scheme reflects the students* est); 
of quality. There will also be criterion-ref e- i;ed tests for each 
unit that should provide useful correlational information. 

Conclusions and Implications 
The word association approach presented nere is far from compl 
is premature to make claims that it will beccne a useful addition t 
evaluator f s assortment of approaches. The scorir^ scheme being emp 
still quite primitive, and further theoretical development is needd 
technique, particularly as it relates to concept development. Howe 
preliminary results have been encouraging. The technique is sensit 
and instructional innovation and it can be applied without substant 
experience. This is a latter of real concern, as many projects cam 
trained ^valuators for all 'if their assessments. Additionally, the 
technique can be <*mployed in evaluation settings vhere objectives < 
particularly well spelled out. An example is the Title I Project m< 
where the major purpose was one of enrichment and thus cognitive out 
not identified or even known in any great detail. 

A final area of potential use for the word association is in pi 
objectives change over time. Many of the stronger evaluation techn: 
demand that project objectives remain the same from beginning to enc 
carefully constructed criterion-ref tranced tests become lefs relevai 
outcomes. Yet in many school projects the process of development i: 
and it is appropriate to shift or alter objectives as the project nu 
The word association lechnique has the advantage of rapidly alterab] 
and no predetermined responses which permits it to be altered with z 
of time and effort expended. 

On the practitioner's level, the technique has been particular] 
for providing feedback to teachers and project personnel. They can 
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changes for each of the concepts used, and relate the findings co instructional 
activities. Revisions in the curriculum, instruction, or measuring instrument 
are easily derived from this feedback procedure. 

For these reasons, we feel that the word association is a promising 
evaluation approach, and we would like to see it explored and critically examined 
by other evaluators. We will be glad to give any details that we have about 
the technique to other evaluators who are interested in applying this technique. 
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Appendix A 

Scoring Scheme for Assessing Quality of Associations of Younger Children 



Score 3 



(a) A superordinate classification 
Examp 1 e : mo th-bug 

horse-animal 
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(b) Description of less important yet applicable function 



Example: car - to carry groceries 
horse - tc pull 
water - to swim in 



Score 2 



(a) A definitive feature or distinguishing 
characteristic of subject 

Example: rain - water 

zoo - animals 

Niagara Falls - water 

(b) Description of the coramon function or 
major use of subject 

Example: car - to carry people 

horse - to ride 

house - to live in 

water - to drink 



Score 1 



Description of less important yet applicable 
use or characteristic 



Example: car - to carry groceries 
horse - to pull 
water - to swim in 



